{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第三章 PyTorch基础：Tensor和Autograd\n",
    "\n",
    "## 3.1 Tensor\n",
    "\n",
    "Tensor，又名张量，读者可能对这个名词似曾相识，因它不仅在PyTorch中出现过，它也是Theano、TensorFlow、\n",
    "Torch和MxNet中重要的数据结构。关于张量的本质不乏深度的剖析，但从工程角度来讲，可简单地认为它就是一个数组，且支持高效的科学计算。它可以是一个数（标量）、一维数组（向量）、二维数组（矩阵）和更高维的数组（高阶数据）。Tensor和Numpy的ndarrays类似，但PyTorch的tensor支持GPU加速。\n",
    "\n",
    "本节将系统讲解tensor的使用，力求面面俱到，但不会涉及每个函数。对于更多函数及其用法，读者可通过在IPython/Notebook中使用函数名加`?`查看帮助文档，或查阅PyTorch官方文档[^1]。\n",
    "\n",
    "[^1]: http://docs.pytorch.org"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.4.0a0+200fb22'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's begin\n",
    "from __future__ import print_function\n",
    "import torch  as t\n",
    "t.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  3.1.1 基础操作\n",
    "\n",
    "学习过Numpy的读者会对本节内容感到非常熟悉，因tensor的接口有意设计成与Numpy类似，以方便用户使用。但不熟悉Numpy也没关系，本节内容并不要求先掌握Numpy。\n",
    "\n",
    "从接口的角度来讲，对tensor的操作可分为两类：\n",
    "\n",
    "1. `torch.function`，如`torch.save`等。\n",
    "2. 另一类是`tensor.function`，如`tensor.view`等。\n",
    "\n",
    "为方便使用，对tensor的大部分操作同时支持这两类接口，在本书中不做具体区分，如`torch.sum (torch.sum(a, b))`与`tensor.sum (a.sum(b))`功能等价。\n",
    "\n",
    "而从存储的角度来讲，对tensor的操作又可分为两类：\n",
    "\n",
    "1. 不会修改自身的数据，如 `a.add(b)`， 加法的结果会返回一个新的tensor。\n",
    "2. 会修改自身的数据，如 `a.add_(b)`， 加法的结果仍存储在a中，a被修改了。\n",
    "\n",
    "函数名以`_`结尾的都是inplace方式, 即会修改调用者自己的数据，在实际应用中需加以区分。\n",
    "\n",
    "#### 创建Tensor\n",
    "\n",
    "在PyTorch中新建tensor的方法有很多，具体如表3-1所示。\n",
    "\n",
    "表3-1: 常见新建tensor的方法\n",
    "\n",
    "|函数|功能|\n",
    "|:---:|:---:|\n",
    "|Tensor(\\*sizes)|基础构造函数|\n",
    "|tensor(data,)|类似np.array的构造函数|\n",
    "|ones(\\*sizes)|全1Tensor|\n",
    "|zeros(\\*sizes)|全0Tensor|\n",
    "|eye(\\*sizes)|对角线为1，其他为0|\n",
    "|arange(s,e,step|从s到e，步长为step|\n",
    "|linspace(s,e,steps)|从s到e，均匀切分成steps份|\n",
    "|rand/randn(\\*sizes)|均匀/标准分布|\n",
    "|normal(mean,std)/uniform(from,to)|正态分布/均匀分布|\n",
    "|randperm(m)|随机排列|\n",
    "\n",
    "这些创建方法都可以在创建的时候指定数据类型dtype和存放device(cpu/gpu).\n",
    "\n",
    "\n",
    "其中使用`Tensor`函数新建tensor是最复杂多变的方式，它既可以接收一个list，并根据list的数据新建tensor，也能根据指定的形状新建tensor，还能传入其他的tensor，下面举几个例子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.00000e-37 *\n",
       "       [[-8.9677,  0.0003, -8.9884],\n",
       "        [ 0.0003,  0.0000,  0.0000]])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 指定tensor的形状\n",
    "a = t.Tensor(2, 3)\n",
    "a # 数值取决于内存空间的状态，print时候可能overflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.,  2.,  3.],\n",
       "        [ 4.,  5.,  6.]])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 用list的数据创建tensor\n",
    "b = t.Tensor([[1,2,3],[4,5,6]])\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.tolist() # 把tensor转为list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`tensor.size()`返回`torch.Size`对象，它是tuple的子类，但其使用方式与tuple略有区别"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 3])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b_size = b.size()\n",
    "b_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.numel() # b中元素总个数，2*3，等价于b.nelement()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(1.00000e-36 *\n",
       "        [[-1.0148,  0.0000,  0.0000],\n",
       "         [ 0.0000,  0.0000,  0.0000]]), tensor([ 2.,  3.]))"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 创建一个和b形状一样的tensor\n",
    "c = t.Tensor(b_size)\n",
    "# 创建一个元素为2和3的tensor\n",
    "d = t.Tensor((2, 3))\n",
    "c, d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "除了`tensor.size()`，还可以利用`tensor.shape`直接查看tensor的形状，`tensor.shape`等价于`tensor.size()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 3])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "需要注意的是，`t.Tensor(*sizes)`创建tensor时，系统不会马上分配空间，只是会计算剩余的内存是否足够使用，使用到tensor时才会分配，而其它操作都是在创建完tensor之后马上进行空间分配。其它常用的创建tensor的方法举例如下。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.,  1.,  1.],\n",
       "        [ 1.,  1.,  1.]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.ones(2, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.,  0.,  0.],\n",
       "        [ 0.,  0.,  0.]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.zeros(2, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 1.,  3.,  5.])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.arange(1, 6, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([  1.0000,   5.5000,  10.0000])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.linspace(1, 10, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.7008, -0.0361, -0.3875],\n",
       "        [ 1.8125, -0.0834, -2.0546]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.randn(2, 3, device=t.device('cpu'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 0,  2,  3,  1,  4])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.randperm(5) # 长度为5的随机排列"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1,  0,  0],\n",
       "        [ 0,  1,  0]], dtype=torch.int32)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.eye(2, 3, dtype=t.int) # 对角线为1, 不要求行列数一致"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`torch.tensor`是在0.4版本新增加的一个新版本的创建tensor方法，使用的方法，和参数几乎和`np.array`完全一致"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scalar: tensor(3.1416), shape of sclar: torch.Size([])\n"
     ]
    }
   ],
   "source": [
    "scalar = t.tensor(3.14159) \n",
    "print('scalar: %s, shape of sclar: %s' %(scalar, scalar.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vector: tensor([ 1,  2]), shape of vector: torch.Size([2])\n"
     ]
    }
   ],
   "source": [
    "vector = t.tensor([1, 2])\n",
    "print('vector: %s, shape of vector: %s' %(vector, vector.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 2])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tensor = t.Tensor(1,2) # 注意和t.tensor([1, 2])的区别\n",
    "tensor.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[ 0.1000,  1.2000],\n",
       "         [ 2.2000,  3.1000],\n",
       "         [ 4.9000,  5.2000]]), torch.Size([3, 2]))"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matrix = t.tensor([[0.1, 1.2], [2.2, 3.1], [4.9, 5.2]])\n",
    "matrix,matrix.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.1111,  0.2222,  0.3333]], dtype=torch.float64)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.tensor([[0.11111, 0.222222, 0.3333333]],\n",
    "                     dtype=t.float64,\n",
    "                     device=t.device('cpu'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([0])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "empty_tensor = t.tensor([])\n",
    "empty_tensor.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 常用Tensor操作"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过`tensor.view`方法可以调整tensor的形状，但必须保证调整前后元素总数一致。`view`不会修改自身的数据，返回的新tensor与源tensor共享内存，也即更改其中的一个，另外一个也会跟着改变。在实际应用中可能经常需要添加或减少某一维度，这时候`squeeze`和`unsqueeze`两个函数就派上用场了。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.,  1.,  2.],\n",
       "        [ 3.,  4.,  5.]])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.arange(0, 6)\n",
    "a.view(2, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 3])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = a.view(-1, 3) # 当某一维为-1的时候，会自动计算它的大小\n",
    "b.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 1, 3])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.unsqueeze(1) # 注意形状，在第1维（下标从0开始）上增加“１” \n",
    "#等价于 b[:,None]\n",
    "b[:, None].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ 0.,  1.,  2.]],\n",
       "\n",
       "        [[ 3.,  4.,  5.]]])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.unsqueeze(-2) # -2表示倒数第二个维度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[[ 0.,  1.,  2.],\n",
       "          [ 3.,  4.,  5.]]]])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = b.view(1, 1, 1, 2, 3)\n",
    "c.squeeze(0) # 压缩第0维的“１”"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.,  1.,  2.],\n",
       "        [ 3.,  4.,  5.]])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c.squeeze() # 把所有维度为“1”的压缩"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[   0.,  100.,    2.],\n",
       "        [   3.,    4.,    5.]])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[1] = 100\n",
    "b # a修改，b作为view之后的，也会跟着修改"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`resize`是另一种可用来调整`size`的方法，但与`view`不同，它可以修改tensor的大小。如果新大小超过了原大小，会自动分配新的内存空间，而如果新大小小于原大小，则之前的数据依旧会被保存，看一个例子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[   0.,  100.,    2.]])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.resize_(1, 3)\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[   0.0000,  100.0000,    2.0000],\n",
       "        [   3.0000,    4.0000,    5.0000],\n",
       "        [  -0.0000,    0.0000,    0.0000]])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.resize_(3, 3) # 旧的数据依旧保存着，多出的大小会分配新空间\n",
    "b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 索引操作\n",
    "\n",
    "Tensor支持与numpy.ndarray类似的索引操作，语法上也类似，下面通过一些例子，讲解常用的索引操作。如无特殊说明，索引出来的结果与原tensor共享内存，也即修改一个，另一个会跟着修改。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.1855, -0.4570,  1.6097, -0.9747],\n",
       "        [-0.5941, -0.8057, -0.6049,  1.5967],\n",
       "        [-0.4694, -1.0633, -0.2432, -0.4794]])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.randn(3, 4)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-0.1855, -0.4570,  1.6097, -0.9747])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[0] # 第0行(下标从0开始)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-0.1855, -0.5941, -0.4694])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[:, 0] # 第0列"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.6097)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[0][2] # 第0行第2个元素，等价于a[0, 2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(-0.9747)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[0, -1] # 第0行最后一个元素"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.1855, -0.4570,  1.6097, -0.9747],\n",
       "        [-0.5941, -0.8057, -0.6049,  1.5967]])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[:2] # 前两行"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.1855, -0.4570],\n",
       "        [-0.5941, -0.8057]])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[:2, 0:2] # 前两行，第0,1列"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.1855, -0.4570]])\n",
      "tensor([-0.1855, -0.4570])\n"
     ]
    }
   ],
   "source": [
    "print(a[0:1, :2]) # 第0行，前两列 \n",
    "print(a[0, :2]) # 注意两者的区别：形状不同"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 3, 4])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# None类似于np.newaxis, 为a新增了一个轴\n",
    "# 等价于a.view(1, a.shape[0], a.shape[1])\n",
    "a[None].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 3, 4])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[None].shape # 等价于a[None,:,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([3, 1, 4])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[:,None,:].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([3, 1, 4, 1, 1])"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[:,None,:,None,None].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0,  0,  1,  0],\n",
       "        [ 0,  0,  0,  1],\n",
       "        [ 0,  0,  0,  0]], dtype=torch.uint8)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a > 1 # 返回一个ByteTensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 1.6097,  1.5967])"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[a>1] # 等价于a.masked_select(a>1)\n",
    "# 选择结果与原tensor不共享内存空间"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.1855, -0.4570,  1.6097, -0.9747],\n",
       "        [-0.5941, -0.8057, -0.6049,  1.5967]])"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[t.LongTensor([0,1])] # 第0行和第1行"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "其它常用的选择函数如表3-2所示。\n",
    "\n",
    "表3-2常用的选择函数\n",
    "\n",
    "函数|功能|\n",
    ":---:|:---:|\n",
    "index_select(input, dim, index)|在指定维度dim上选取，比如选取某些行、某些列\n",
    "masked_select(input, mask)|例子如上，a[a>0]，使用ByteTensor进行选取\n",
    "non_zero(input)|非0元素的下标\n",
    "gather(input, dim, index)|根据index，在dim维度上选取数据，输出的size与index一样\n",
    "\n",
    "\n",
    "`gather`是一个比较复杂的操作，对一个2维tensor，输出的每个元素如下：\n",
    "\n",
    "```python\n",
    "out[i][j] = input[index[i][j]][j]  # dim=0\n",
    "out[i][j] = input[i][index[i][j]]  # dim=1\n",
    "```\n",
    "三维tensor的`gather`操作同理，下面举几个例子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  0.,   1.,   2.,   3.],\n",
       "        [  4.,   5.,   6.,   7.],\n",
       "        [  8.,   9.,  10.,  11.],\n",
       "        [ 12.,  13.,  14.,  15.]])"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.arange(0, 16).view(4, 4)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  0.,   5.,  10.,  15.]])"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 选取对角线的元素\n",
    "index = t.LongTensor([[0,1,2,3]])\n",
    "a.gather(0, index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  3.],\n",
       "        [  6.],\n",
       "        [  9.],\n",
       "        [ 12.]])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 选取反对角线上的元素\n",
    "index = t.LongTensor([[3,2,1,0]]).t()\n",
    "a.gather(1, index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 12.,   9.,   6.,   3.]])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 选取反对角线上的元素，注意与上面的不同\n",
    "index = t.LongTensor([[3,2,1,0]])\n",
    "a.gather(0, index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  0.,   3.],\n",
       "        [  5.,   6.],\n",
       "        [ 10.,   9.],\n",
       "        [ 15.,  12.]])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 选取两个对角线上的元素\n",
    "index = t.LongTensor([[0,1,2,3],[3,2,1,0]]).t()\n",
    "b = a.gather(1, index)\n",
    "b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "与`gather`相对应的逆操作是`scatter_`，`gather`把数据从input中按index取出，而`scatter_`是把取出的数据再放回去。注意`scatter_`函数是inplace操作。\n",
    "\n",
    "```python\n",
    "out = input.gather(dim, index)\n",
    "-->近似逆操作\n",
    "out = Tensor()\n",
    "out.scatter_(dim, index)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  0.,   0.,   0.,   3.],\n",
       "        [  0.,   5.,   6.,   0.],\n",
       "        [  0.,   9.,  10.,   0.],\n",
       "        [ 12.,   0.,   0.,  15.]])"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 把两个对角线元素放回去到指定位置\n",
    "c = t.zeros(4,4)\n",
    "c.scatter_(1, index, b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对tensor的任何索引操作仍是一个tensor，想要获取标准的python对象数值，需要调用`tensor.item()`, 这个方法只对包含一个元素的tensor适用"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[0,0] #依旧是tensor）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[0,0].item() # python float"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 1, 1])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = a[0:1, 0:1, None]\n",
    "print(d.shape)\n",
    "d.item() # 只包含一个元素的tensor即可调用tensor.item,与形状无关"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "# a[0].item()  ->\n",
    "# raise ValueError: only one element tensors can be converted to Python scalars"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 高级索引\n",
    "PyTorch在0.2版本中完善了索引操作，目前已经支持绝大多数numpy的高级索引[^10]。高级索引可以看成是普通索引操作的扩展，但是高级索引操作的结果一般不和原始的Tensor贡献内出。 \n",
    "[^10]: https://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#advanced-indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[  0.,   1.,   2.],\n",
       "         [  3.,   4.,   5.],\n",
       "         [  6.,   7.,   8.]],\n",
       "\n",
       "        [[  9.,  10.,  11.],\n",
       "         [ 12.,  13.,  14.],\n",
       "         [ 15.,  16.,  17.]],\n",
       "\n",
       "        [[ 18.,  19.,  20.],\n",
       "         [ 21.,  22.,  23.],\n",
       "         [ 24.,  25.,  26.]]])"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = t.arange(0,27).view(3,3,3)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 14.,  24.])"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[[1, 2], [1, 2], [2, 0]] # x[1,1,2]和x[2,2,0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 19.,  10.,   1.])"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[[2, 1, 0], [0], [1]] # x[2,0,1],x[1,0,1],x[0,0,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[  0.,   1.,   2.],\n",
       "         [  3.,   4.,   5.],\n",
       "         [  6.,   7.,   8.]],\n",
       "\n",
       "        [[ 18.,  19.,  20.],\n",
       "         [ 21.,  22.,  23.],\n",
       "         [ 24.,  25.,  26.]]])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[[0, 2], ...] # x[0] 和 x[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tensor类型\n",
    "\n",
    "Tensor有不同的数据类型，如表3-3所示，每种类型分别对应有CPU和GPU版本(HalfTensor除外)。默认的tensor是FloatTensor，可通过`t.set_default_tensor_type` 来修改默认tensor类型(如果默认类型为GPU tensor，则所有操作都将在GPU上进行)。Tensor的类型对分析内存占用很有帮助。例如对于一个size为(1000, 1000, 1000)的FloatTensor，它有`1000*1000*1000=10^9`个元素，每个元素占32bit/8 = 4Byte内存，所以共占大约4GB内存/显存。HalfTensor是专门为GPU版本设计的，同样的元素个数，显存占用只有FloatTensor的一半，所以可以极大缓解GPU显存不足的问题，但由于HalfTensor所能表示的数值大小和精度有限[^2]，所以可能出现溢出等问题。\n",
    "\n",
    "[^2]: https://stackoverflow.com/questions/872544/what-range-of-numbers-can-be-represented-in-a-16-32-and-64-bit-ieee-754-syste\n",
    "\n",
    "表3-3: tensor数据类型\n",
    "\n",
    "| Data type                | dtype                             | CPU tensor                                                   | GPU tensor                |\n",
    "| ------------------------ | --------------------------------- | ------------------------------------------------------------ | ------------------------- |\n",
    "| 32-bit floating point    | `torch.float32` or `torch.float`  | `torch.FloatTensor`                                          | `torch.cuda.FloatTensor`  |\n",
    "| 64-bit floating point    | `torch.float64` or `torch.double` | `torch.DoubleTensor`                                         | `torch.cuda.DoubleTensor` |\n",
    "| 16-bit floating point    | `torch.float16` or `torch.half`   | `torch.HalfTensor`                                           | `torch.cuda.HalfTensor`   |\n",
    "| 8-bit integer (unsigned) | `torch.uint8`                     | [`torch.ByteTensor`](https://pytorch.org/docs/stable/tensors.html#torch.ByteTensor) | `torch.cuda.ByteTensor`   |\n",
    "| 8-bit integer (signed)   | `torch.int8`                      | `torch.CharTensor`                                           | `torch.cuda.CharTensor`   |\n",
    "| 16-bit integer (signed)  | `torch.int16` or `torch.short`    | `torch.ShortTensor`                                          | `torch.cuda.ShortTensor`  |\n",
    "| 32-bit integer (signed)  | `torch.int32` or `torch.int`      | `torch.IntTensor`                                            | `torch.cuda.IntTensor`    |\n",
    "| 64-bit integer (signed)  | `torch.int64` or `torch.long`     | `torch.LongTensor`                                           | `torch.cuda.LongTensor`   |\n",
    "\n",
    " \n",
    "\n",
    "各数据类型之间可以互相转换，`type(new_type)`是通用的做法，同时还有`float`、`long`、`half`等快捷方法。CPU tensor与GPU tensor之间的互相转换通过`tensor.cuda`和`tensor.cpu`方法实现，此外还可以使用`tensor.to(device)`。Tensor还有一个`new`方法，用法与`t.Tensor`一样，会调用该tensor对应类型的构造函数，生成与当前tensor类型一致的tensor。`torch.*_like(tensora)` 可以生成和`tensora`拥有同样属性(类型，形状，cpu/gpu)的新tensor。 `tensor.new_*(new_shape)` 新建一个不同形状的tensor。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 设置默认tensor，注意参数是字符串\n",
    "t.set_default_tensor_type('torch.DoubleTensor')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.float64"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.Tensor(2,3)\n",
    "a.dtype # 现在a是DoubleTensor,dtype是float64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 恢复之前的默认设置\n",
    "t.set_default_tensor_type('torch.FloatTensor')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.float32"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 把a转成FloatTensor，等价于b=a.type(t.FloatTensor)\n",
    "b = a.float() \n",
    "b.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.,  0.,  0.],\n",
       "        [ 0.,  0.,  0.]])"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = a.type_as(b)\n",
    "c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 4.6390e-310,  1.4147e+161,   1.4917e+20],\n",
       "        [ 2.0093e+174,  1.4327e+228,  1.3404e-317]], dtype=torch.float64)"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.new(2,3) # 等价于torch.DoubleTensor(2,3)，建议使用a.new_tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.,  0.,  0.],\n",
       "        [ 0.,  0.,  0.]], dtype=torch.float64)"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.zeros_like(a) #等价于t.zeros(a.shape,dtype=a.dtype,device=a.device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0,  0,  0],\n",
       "        [ 0,  0,  0]], dtype=torch.int16)"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.zeros_like(a, dtype=t.int16) #可以修改某些属性"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.8882,  0.7037,  0.5756],\n",
       "        [ 0.7113,  0.2868,  0.7597]], dtype=torch.float64)"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.rand_like(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1,  1,  1,  1,  1],\n",
       "        [ 1,  1,  1,  1,  1],\n",
       "        [ 1,  1,  1,  1,  1],\n",
       "        [ 1,  1,  1,  1,  1]], dtype=torch.int32)"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.new_ones(4,5, dtype=t.int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 3.,  4.], dtype=torch.float64)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.new_tensor([3,4]) # "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 逐元素操作\n",
    "\n",
    "这部分操作会对tensor的每一个元素(point-wise，又名element-wise)进行操作，此类操作的输入与输出形状一致。常用的操作如表3-4所示。\n",
    "\n",
    "表3-4: 常见的逐元素操作\n",
    "\n",
    "|函数|功能|\n",
    "|:--:|:--:|\n",
    "|abs/sqrt/div/exp/fmod/log/pow..|绝对值/平方根/除法/指数/求余/求幂..|\n",
    "|cos/sin/asin/atan2/cosh..|相关三角函数|\n",
    "|ceil/round/floor/trunc| 上取整/四舍五入/下取整/只保留整数部分|\n",
    "|clamp(input, min, max)|超过min和max部分截断|\n",
    "|sigmod/tanh..|激活函数\n",
    "\n",
    "对于很多操作，例如div、mul、pow、fmod等，PyTorch都实现了运算符重载，所以可以直接使用运算符。如`a ** 2` 等价于`torch.pow(a,2)`, `a * 2`等价于`torch.mul(a,2)`。\n",
    "\n",
    "其中`clamp(x, min, max)`的输出满足以下公式：\n",
    "$$\n",
    "y_i =\n",
    "\\begin{cases}\n",
    "min,  & \\text{if  } x_i \\lt min \\\\\n",
    "x_i,  & \\text{if  } min \\le x_i \\le max  \\\\\n",
    "max,  & \\text{if  } x_i \\gt max\\\\\n",
    "\\end{cases}\n",
    "$$\n",
    "`clamp`常用在某些需要比较大小的地方，如取一个tensor的每个元素与另一个数的较大值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.0000000000, 0.5403022766, -0.4161468446],\n",
       "        [-0.9899924994, -0.6536436081, 0.2836622000]])"
      ]
     },
     "execution_count": 157,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.arange(0, 6).view(2, 3)\n",
    "t.cos(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.,  1.,  2.],\n",
       "        [ 0.,  1.,  2.]])"
      ]
     },
     "execution_count": 158,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a % 3 # 等价于t.fmod(a, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  0.,   1.,   4.],\n",
       "        [  9.,  16.,  25.]])"
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a ** 2 # 等价于t.pow(a, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0.,  1.,  2.],\n",
      "        [ 3.,  4.,  5.]])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[ 3.,  3.,  3.],\n",
       "        [ 3.,  4.,  5.]])"
      ]
     },
     "execution_count": 160,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 取a中的每一个元素与3相比较大的一个 (小于3的截断成3)\n",
    "print(a)\n",
    "t.clamp(a, min=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.0000000000, 0.8414709568, 0.9092974067],\n",
       "        [0.1411200017, -0.7568024993, -0.9589242935]])"
      ]
     },
     "execution_count": 162,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = a.sin_() # 效果同 a = a.sin();b=a ,但是更高效节省显存\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  归并操作 \n",
    "此类操作会使输出形状小于输入形状，并可以沿着某一维度进行指定操作。如加法`sum`，既可以计算整个tensor的和，也可以计算tensor中每一行或每一列的和。常用的归并操作如表3-5所示。\n",
    "\n",
    "表3-5: 常用归并操作\n",
    "\n",
    "|函数|功能|\n",
    "|:---:|:---:|\n",
    "|mean/sum/median/mode|均值/和/中位数/众数|\n",
    "|norm/dist|范数/距离|\n",
    "|std/var|标准差/方差|\n",
    "|cumsum/cumprod|累加/累乘|\n",
    "\n",
    "以上大多数函数都有一个参数**`dim`**，用来指定这些操作是在哪个维度上执行的。关于dim(对应于Numpy中的axis)的解释众说纷纭，这里提供一个简单的记忆方式：\n",
    "\n",
    "假设输入的形状是(m, n, k)\n",
    "\n",
    "- 如果指定dim=0，输出的形状就是(1, n, k)或者(n, k)\n",
    "- 如果指定dim=1，输出的形状就是(m, 1, k)或者(m, k)\n",
    "- 如果指定dim=2，输出的形状就是(m, n, 1)或者(m, n)\n",
    "\n",
    "size中是否有\"1\"，取决于参数`keepdim`，`keepdim=True`会保留维度`1`。注意，以上只是经验总结，并非所有函数都符合这种形状变化方式，如`cumsum`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 2.,  2.,  2.]])"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = t.ones(2, 3)\n",
    "b.sum(dim = 0, keepdim=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 2.,  2.,  2.])"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# keepdim=False，不保留维度\"1\"，注意形状\n",
    "b.sum(dim=0, keepdim=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 3.,  3.])"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.sum(dim=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0.,  1.,  2.],\n",
      "        [ 3.,  4.,  5.]])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[  0.,   1.,   3.],\n",
       "        [  3.,   7.,  12.]])"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.arange(0, 6).view(2, 3)\n",
    "print(a)\n",
    "a.cumsum(dim=1) # 沿着行累加"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 比较\n",
    "比较函数中有一些是逐元素比较，操作类似于逐元素操作，还有一些则类似于归并操作。常用比较函数如表3-6所示。\n",
    "\n",
    "表3-6: 常用比较函数\n",
    "\n",
    "|函数|功能|\n",
    "|:--:|:--:|\n",
    "|gt/lt/ge/le/eq/ne|大于/小于/大于等于/小于等于/等于/不等|\n",
    "|topk|最大的k个数|\n",
    "|sort|排序|\n",
    "|max/min|比较两个tensor最大最小值|\n",
    "\n",
    "表中第一行的比较操作已经实现了运算符重载，因此可以使用`a>=b`、`a>b`、`a!=b`、`a==b`，其返回结果是一个`ByteTensor`，可用来选取元素。max/min这两个操作比较特殊，以max来说，它有以下三种使用情况：\n",
    "- t.max(tensor)：返回tensor中最大的一个数\n",
    "- t.max(tensor,dim)：指定维上最大的数，返回tensor和下标\n",
    "- t.max(tensor1, tensor2): 比较两个tensor相比较大的元素\n",
    "\n",
    "至于比较一个tensor和一个数，可以使用clamp函数。下面举例说明。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  0.,   3.,   6.],\n",
       "        [  9.,  12.,  15.]])"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.linspace(0, 15, 6).view(2, 3)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 15.,  12.,   9.],\n",
       "        [  6.,   3.,   0.]])"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = t.linspace(15, 0, 6).view(2, 3)\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0,  0,  0],\n",
       "        [ 1,  1,  1]], dtype=torch.uint8)"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a>b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([  9.,  12.,  15.])"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[a>b] # a中大于b的元素"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(15.)"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.max(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([ 15.,   6.]), tensor([ 0,  0]))"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.max(b, dim=1) \n",
    "# 第一个返回值的15和6分别表示第0行和第1行最大的元素\n",
    "# 第二个返回值的0和0表示上述最大的数是该行第0个元素"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 15.,  12.,   9.],\n",
       "        [  9.,  12.,  15.]])"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.max(a,b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 10.,  10.,  10.],\n",
       "        [ 10.,  12.,  15.]])"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 比较a和10较大的元素\n",
    "t.clamp(a, min=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 线性代数\n",
    "\n",
    "PyTorch的线性函数主要封装了Blas和Lapack，其用法和接口都与之类似。常用的线性代数函数如表3-7所示。\n",
    "\n",
    "表3-7: 常用的线性代数函数\n",
    "\n",
    "|函数|功能|\n",
    "|:---:|:---:|\n",
    "|trace|对角线元素之和(矩阵的迹)|\n",
    "|diag|对角线元素|\n",
    "|triu/tril|矩阵的上三角/下三角，可指定偏移量|\n",
    "|mm/bmm|矩阵乘法，batch的矩阵乘法|\n",
    "|addmm/addbmm/addmv/addr/badbmm..|矩阵运算\n",
    "|t|转置|\n",
    "|dot/cross|内积/外积\n",
    "|inverse|求逆矩阵\n",
    "|svd|奇异值分解\n",
    "\n",
    "具体使用说明请参见官方文档[^3]，需要注意的是，矩阵的转置会导致存储空间不连续，需调用它的`.contiguous`方法将其转为连续。\n",
    "[^3]: http://pytorch.org/docs/torch.html#blas-and-lapack-operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = a.t()\n",
    "b.is_contiguous()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  0.,   9.],\n",
       "        [  3.,  12.],\n",
       "        [  6.,  15.]])"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.contiguous()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1.2 Tensor和Numpy\n",
    "\n",
    "Tensor和Numpy数组之间具有很高的相似性，彼此之间的互操作也非常简单高效。需要注意的是，Numpy和Tensor共享内存。由于Numpy历史悠久，支持丰富的操作，所以当遇到Tensor不支持的操作时，可先转成Numpy数组，处理后再转回tensor，其转换开销很小。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 1., 1.],\n",
       "       [1., 1., 1.]], dtype=float32)"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "a = np.ones([2, 3],dtype=np.float32)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.,  1.,  1.],\n",
       "        [ 1.,  1.,  1.]])"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = t.from_numpy(a)\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.,  1.,  1.],\n",
       "        [ 1.,  1.,  1.]])"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = t.Tensor(a) # 也可以直接将numpy对象传入Tensor\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[   1.,  100.,    1.],\n",
       "        [   1.,    1.,    1.]])"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[0, 1]=100\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  1., 100.,   1.],\n",
       "       [  1.,   1.,   1.]], dtype=float32)"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = b.numpy() # a, b, c三个对象共享内存\n",
    "c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**注意**： 当numpy的数据类型和Tensor的类型不一样的时候，数据会被复制，不会共享内存。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('float64')"
      ]
     },
     "execution_count": 152,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = np.ones([2, 3])\n",
    "# 注意和上面的a的区别（dtype不是float32）\n",
    "a.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.float32"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = t.Tensor(a) # 此处进行拷贝，不共享内存\n",
    "b.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.,  1.,  1.],\n",
       "        [ 1.,  1.,  1.]], dtype=torch.float64)"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = t.from_numpy(a) # 注意c的类型（DoubleTensor）\n",
    "c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.,  1.,  1.],\n",
       "        [ 1.,  1.,  1.]])"
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[0, 1] = 100\n",
    "b # b与a不共享内存，所以即使a改变了，b也不变"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[   1.,  100.,    1.],\n",
       "        [   1.,    1.,    1.]], dtype=torch.float64)"
      ]
     },
     "execution_count": 156,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c # c与a共享内存"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**注意：** 不论输入的类型是什么，t.tensor都会进行数据拷贝，不会共享内存"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "tensor = t.tensor(a) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  1., 100.,   1.],\n",
       "       [  1.,   1.,   1.]])"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tensor[0,0]=0\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "广播法则(broadcast)是科学运算中经常使用的一个技巧，它在快速执行向量化的同时不会占用额外的内存/显存。\n",
    "Numpy的广播法则定义如下：\n",
    "\n",
    "- 让所有输入数组都向其中shape最长的数组看齐，shape中不足的部分通过在前面加1补齐\n",
    "- 两个数组要么在某一个维度的长度一致，要么其中一个为1，否则不能计算 \n",
    "- 当输入数组的某个维度的长度为1时，计算时沿此维度复制扩充成一样的形状\n",
    "\n",
    "PyTorch当前已经支持了自动广播法则，但是笔者还是建议读者通过以下两个函数的组合手动实现广播法则，这样更直观，更不易出错：\n",
    "\n",
    "- `unsqueeze`或者`view`，或者tensor[None],：为数据某一维的形状补1，实现法则1\n",
    "- `expand`或者`expand_as`，重复数组，实现法则3；该操作不会复制数组，所以不会占用额外的空间。\n",
    "\n",
    "注意，repeat实现与expand相类似的功能，但是repeat会把相同数据复制多份，因此会占用额外的空间。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "a = t.ones(3, 2)\n",
    "b = t.zeros(2, 3,1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ 1.,  1.],\n",
       "         [ 1.,  1.],\n",
       "         [ 1.,  1.]],\n",
       "\n",
       "        [[ 1.,  1.],\n",
       "         [ 1.,  1.],\n",
       "         [ 1.,  1.]]])"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 自动广播法则\n",
    "# 第一步：a是2维,b是3维，所以先在较小的a前面补1 ，\n",
    "#               即：a.unsqueeze(0)，a的形状变成（1，3，2），b的形状是（2，3，1）,\n",
    "# 第二步:   a和b在第一维和第三维形状不一样，其中一个为1 ，\n",
    "#               可以利用广播法则扩展，两个形状都变成了（2，3，2）\n",
    "a+b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ 1.,  1.],\n",
       "         [ 1.,  1.],\n",
       "         [ 1.,  1.]],\n",
       "\n",
       "        [[ 1.,  1.],\n",
       "         [ 1.,  1.],\n",
       "         [ 1.,  1.]]])"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 手动广播法则\n",
    "# 或者 a.view(1,3,2).expand(2,3,2)+b.expand(2,3,2)\n",
    "a[None].expand(2, 3, 2) + b.expand(2,3,2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "# expand不会占用额外空间，只会在需要的时候才扩充，可极大节省内存\n",
    "e = a.unsqueeze(0).expand(10000000000000, 3,2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1.3 内部结构\n",
    "\n",
    "tensor的数据结构如图3-1所示。tensor分为头信息区(Tensor)和存储区(Storage)，信息区主要保存着tensor的形状（size）、步长（stride）、数据类型（type）等信息，而真正的数据则保存成连续数组。由于数据动辄成千上万，因此信息区元素占用内存较少，主要内存占用则取决于tensor中元素的数目，也即存储区的大小。\n",
    "\n",
    "一般来说一个tensor有着与之相对应的storage, storage是在data之上封装的接口，便于使用，而不同tensor的头信息一般不同，但却可能使用相同的数据。下面看两个例子。\n",
    "\n",
    "![图3-1: Tensor的数据结构](imgs/tensor_data_structure.svg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       " 0.0\n",
       " 1.0\n",
       " 2.0\n",
       " 3.0\n",
       " 4.0\n",
       " 5.0\n",
       "[torch.FloatStorage of size 6]"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.arange(0, 6)\n",
    "a.storage()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       " 0.0\n",
       " 1.0\n",
       " 2.0\n",
       " 3.0\n",
       " 4.0\n",
       " 5.0\n",
       "[torch.FloatStorage of size 6]"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = a.view(2, 3)\n",
    "b.storage()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 一个对象的id值可以看作它在内存中的地址\n",
    "# storage的内存地址一样，即是同一个storage\n",
    "id(b.storage()) == id(a.storage())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[   0.,  100.,    2.],\n",
       "        [   3.,    4.,    5.]])"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# a改变，b也随之改变，因为他们共享storage\n",
    "a[1] = 100\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       " 0.0\n",
       " 100.0\n",
       " 2.0\n",
       " 3.0\n",
       " 4.0\n",
       " 5.0\n",
       "[torch.FloatStorage of size 6]"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = a[2:] \n",
    "c.storage()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(93894489135160, 93894489135152)"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c.data_ptr(), a.data_ptr() # data_ptr返回tensor首元素的内存地址\n",
    "# 可以看出相差8，这是因为2*4=8--相差两个元素，每个元素占4个字节(float)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([   0.,  100., -100.,    3.,    4.,    5.])"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c[0] = -100 # c[0]的内存地址对应a[2]的内存地址\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 6666.,   100.,  -100.],\n",
       "        [    3.,     4.,     5.]])"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = t.Tensor(c.storage())\n",
    "d[0] = 6666\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 下面４个tensor共享storage\n",
    "id(a.storage()) == id(b.storage()) == id(c.storage()) == id(d.storage())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 2, 0)"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.storage_offset(), c.storage_offset(), d.storage_offset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e = b[::2, ::2] # 隔2行/列取一个元素\n",
    "id(e.storage()) == id(a.storage())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((3, 1), (6, 2))"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.stride(), e.stride()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e.is_contiguous()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可见绝大多数操作并不修改tensor的数据，而只是修改了tensor的头信息。这种做法更节省内存，同时提升了处理速度。在使用中需要注意。\n",
    "此外有些操作会导致tensor不连续，这时需调用`tensor.contiguous`方法将它们变成连续的数据，该方法会使数据复制一份，不再与原来的数据共享storage。\n",
    "另外读者可以思考一下，之前说过的高级索引一般不共享stroage，而普通索引共享storage，这是为什么？（提示：普通索引可以通过只修改tensor的offset，stride和size，而不修改storage来实现）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1.4 其它有关Tensor的话题\n",
    "这部分的内容不好专门划分一小节，但是笔者认为仍值得读者注意，故而将其放在这一小节。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### GPU/CPU\n",
    "tensor可以很随意的在gpu/cpu上传输。使用`tensor.cuda(device_id)`或者`tensor.cpu()`。另外一个更通用的方法是`tensor.to(device)`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cpu')"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.randn(3, 4)\n",
    "a.device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "if t.cuda.is_available():\n",
    "    a = t.randn(3,4, device=t.device('cuda:1'))\n",
    "    # 等价于\n",
    "    # a.t.randn(3,4).cuda(1)\n",
    "    # 但是前者更快\n",
    "    a.device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.4362, -0.4324, -0.3985,  0.2730],\n",
       "        [-0.1356, -0.3448,  1.5713, -0.8333],\n",
       "        [-0.0089,  1.4516, -0.2062, -0.4085]])"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "device = t.device('cpu')\n",
    "a.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**注意**\n",
    "- 尽量使用`tensor.to(device)`, 将`device`设为一个可配置的参数，这样可以很轻松的使程序同时兼容GPU和CPU\n",
    "- 数据在GPU之中传输的速度要远快于内存(CPU)到显存(GPU), 所以尽量避免频繁的在内存和显存中传输数据。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 持久化\n",
    "Tensor的保存和加载十分的简单，使用t.save和t.load即可完成相应的功能。在save/load时可指定使用的`pickle`模块，在load时还可将GPU tensor映射到CPU或其它GPU上。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "if t.cuda.is_available():\n",
    "    a = a.cuda(1) # 把a转为GPU1上的tensor,\n",
    "    t.save(a,'a.pth')\n",
    "\n",
    "    # 加载为b, 存储于GPU1上(因为保存时tensor就在GPU1上)\n",
    "    b = t.load('a.pth')\n",
    "    # 加载为c, 存储于CPU\n",
    "    c = t.load('a.pth', map_location=lambda storage, loc: storage)\n",
    "    # 加载为d, 存储于GPU0上\n",
    "    d = t.load('a.pth', map_location={'cuda:1':'cuda:0'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####   向量化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "向量化计算是一种特殊的并行计算方式，相对于一般程序在同一时间只执行一个操作的方式，它可在同一时间执行多个操作，通常是对不同的数据执行同样的一个或一批指令，或者说把指令应用于一个数组/向量上。向量化可极大提高科学运算的效率，Python本身是一门高级语言，使用很方便，但这也意味着很多操作很低效，尤其是`for`循环。在科学计算程序中应当极力避免使用Python原生的`for循环`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "def for_loop_add(x, y):\n",
    "    result = []\n",
    "    for i,j in zip(x, y):\n",
    "        result.append(i + j)\n",
    "    return t.Tensor(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "777 µs ± 17 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)\n",
      "The slowest run took 10.00 times longer than the fastest. This could mean that an intermediate result is being cached.\n",
      "6.2 µs ± 8.48 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
     ]
    }
   ],
   "source": [
    "x = t.zeros(100)\n",
    "y = t.ones(100)\n",
    "%timeit -n 10 for_loop_add(x, y)\n",
    "%timeit -n 10 x + y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可见二者有超过几十倍的速度差距，因此在实际使用中应尽量调用内建函数(buildin-function)，这些函数底层由C/C++实现，能通过执行底层优化实现高效计算。因此在平时写代码时，就应养成向量化的思维习惯，千万避免对较大的tensor进行逐元素遍历。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此外还有以下几点需要注意：\n",
    "- 大多数`t.function`都有一个参数`out`，这时候产生的结果将保存在out指定tensor之中。\n",
    "- `t.set_num_threads`可以设置PyTorch进行CPU多线程并行计算时候所占用的线程数，这个可以用来限制PyTorch所占用的CPU数目。\n",
    "- `t.set_printoptions`可以用来设置打印tensor时的数值精度和格式。\n",
    "下面举例说明。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(1.6777e+07) tensor(1.6777e+07)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(tensor(2.0000e+07), tensor(2.0000e+07))"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.arange(0, 20000000)\n",
    "print(a[-1], a[-2]) # 32bit的IntTensor精度有限导致溢出\n",
    "b = t.LongTensor()\n",
    "t.arange(0, 20000000, out=b) # 64bit的LongTensor不会溢出\n",
    "b[-1],b[-2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.4317,  2.1251,  1.0684],\n",
       "        [ 1.0297,  1.0137, -0.8691]])"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.randn(2,3)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.4316843748, 2.1250586510, 1.0684313774],\n",
       "        [1.0297036171, 1.0137003660, -0.8690901995]])"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.set_printoptions(precision=10)\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1.5 小试牛刀：线性回归"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "线性回归是机器学习入门知识，应用十分广泛。线性回归利用数理统计中回归分析，来确定两种或两种以上变量间相互依赖的定量关系的，其表达形式为$y = wx+b+e$，$e$为误差服从均值为0的正态分布。首先让我们来确认线性回归的损失函数：\n",
    "$$\n",
    "loss = \\sum_i^N \\frac 1 2 ({y_i-(wx_i+b)})^2\n",
    "$$\n",
    "然后利用随机梯度下降法更新参数$\\textbf{w}$和$\\textbf{b}$来最小化损失函数，最终学得$\\textbf{w}$和$\\textbf{b}$的数值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch as t\n",
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt\n",
    "from IPython import display\n",
    "\n",
    "device = t.device('cpu') #如果你想用gpu，改成t.device('cuda:0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 设置随机数种子，保证在不同电脑上运行时下面的输出一致\n",
    "t.manual_seed(1000) \n",
    "\n",
    "def get_fake_data(batch_size=8):\n",
    "    ''' 产生随机数据：y=x*2+3，加上了一些噪声'''\n",
    "    x = t.rand(batch_size, 1, device=device) * 5\n",
    "    y = x * 2 + 3 +  t.randn(batch_size, 1, device=device)\n",
    "    return x, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x7fb5f2e5f780>"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAEKpJREFUeJzt3X9s3Pddx/HXC8ddr92GUXNA47Zk\nkybzo6F1sKKWimqsBXejar3QP4K0QSdBEAzW8ofRwh9M458IGSF+SUxhHSowuo0utUq11i0aBfFH\nM9y6XVJSQ1fWredBrhtuV3pqE/PmD59d+3L2fS++7933c/d8SFbO52+c1zff5HVff76fz30dEQIA\npON7eh0AANAeihsAEkNxA0BiKG4ASAzFDQCJobgBIDEUNwAkhuIGgMRQ3ACQmF15fNPdu3fH3r17\n8/jWANCXnnzyyZcjopxl21yKe+/evZqfn8/jWwNAX7L9YtZtGSoBgMRQ3ACQGIobABJDcQNAYihu\nAEgMxQ0AicllOiAADIrZhYpm5ha1tFzTnpGSpifHNDU+muufSXEDwAWaXajoyPGTqp1dkSRVlms6\ncvykJOVa3gyVAMAFmplbXC/tNbWzK5qZW8z1z6W4AeACLS3X2nq+UyhuALhAe0ZKbT3fKRQ3AFyg\n6ckxlYaHNj1XGh7S9ORYrn8uFycB4AKtXYBkVgkAJGRqfDT3om7EUAkAJIbiBoDEUNwAkBjGuAEU\nVi+Wk6cg0xm37btsn7L9rO278w4FAGvLySvLNYXeWk4+u1DpdbSea1nctq+W9CuSDki6RtKttt+T\ndzAAg61Xy8lTkOWM+0ckPRERr0fEOUn/JOmD+cYCMOh6tZw8BVmK+5SkG21fZvsSSR+QdGW+sQAM\nul4tJ09By+KOiNOSfl/SY5IekfSMpHON29k+bHve9ny1Wu14UACDpVfLyVOQ6eJkRNwTEfsj4kZJ\n35H0H022ORYRExExUS6XO50TwICZGh/V0YP7NDpSkiWNjpR09OA+ZpUo43RA298fEWdsXyXpoKTr\n840FAK2Xkw/qdMGs87i/aPsySWclfTQi/ifHTADQUq/uPlMEmYo7In4q7yAA0I7tpgv2e3Gz5B1A\nkgZ5uiDFDSBJgzxdkOIGkKRBni7Im0wBSFKv7j5TBBQ3gGT14u4zRUBxA+g7/T6/m+IG0FcGYX43\nFycB9JVBeDtYihtAXxmE+d0UN4C+MgjzuyluAH1lEOZ3c3ESQF8ZhPndFDeAvtPv87sZKgGAxFDc\nAJAYihsAEkNxA0BiKG4ASAzFDQCJobgBIDEUNwAkJlNx2/4t28/aPmX7PtsX5x0MANBcy+K2PSrp\nY5ImIuJqSUOSDuUdDADQXNahkl2SSrZ3SbpE0lJ+kQAA22n5XiURUbH9B5K+Iakm6dGIeLRxO9uH\nJR2WpKuuuqrTOQE00e+36EJzWYZKvk/S7ZLeJWmPpEttf6hxu4g4FhETETFRLpc7nxTAJmu36Kos\n1xR66xZdswuVXkdDzrIMldws6T8johoRZyUdl/ST+cYC0Mog3KILzWUp7m9Ius72JbYt6SZJp/ON\nBaCVQbhFF5prWdwRcULS/ZKeknSy/nuO5ZwLQAuDcIsuNJdpVklEfCIifjgiro6ID0fEG3kHA7C9\nQbhFF5rjDjhAogbhFl1ojuIGEtbvt+hCc7xXCQAkhuIGgMQwVAJ0GasdsVMUN9BFa6sd1xbOrK12\nlER5IzOKG30hlbPY7VY7FjEvioniRvJSOotltSM6gYuTSF5K79nBakd0AsWN5KV0FstqR3QCQyVI\n3p6RkipNSrrbZ7FZxtlZ7YhOoLiRvOnJsU1j3FL3z2LbGWdntSN2iqESJG9qfFRHD+7T6EhJljQ6\nUtLRg/u6Wo4pjbMjfZxxoy/0+iw2pXF2pI8zbqADmC2CbqK4gQ5gtgi6iaESoAOYLYJuoriBDun1\nODsGB0MlAJAYihsAEtOyuG2P2X56w8ertu/uRjgAwPlajnFHxKKkayXJ9pCkiqQHcs4FANhCu0Ml\nN0n6WkS8mEcYAEBr7Rb3IUn3NfuC7cO2523PV6vVnScDADSVubhtXyTpNkl/1+zrEXEsIiYiYqJc\nLncqHwCgQTtn3O+X9FRE/HdeYQAArbVT3L+gLYZJAADdk2nlpO1LJP2MpF/NNw7Q/1K5sTGKK1Nx\nR8Trki7LOQvQ91K6sTGKi/cqQV8r2tntdjdcoLiRFcWNvlXEs1tuuIBO4L1K0LeKeDsxbriATqC4\n0beKeHbLDRfQCRQ3+lYRz26LcGNjpI8xbvSt6cmxTWPcUjHObrnhAnaK4kbf4nZi6FcUN/oaZ7fo\nR4xxA0BiKG4ASAzFDQCJobgBIDEUNwAkhuIGgMRQ3ACQGIobABJDcQNAYihuAEgMxQ0AiaG4ASAx\nmYrb9ojt+20/Z/u07evzDgYAaC7ruwP+saRHIuIO2xdJuiTHTACAbbQsbtvvlHSjpDslKSLelPRm\nvrEAAFvJMlTybklVSX9pe8H2p21fmnMuAMAWshT3Lkn7Jf15RIxL+l9JH2/cyPZh2/O256vVaodj\nAgDWZCnulyS9FBEn6p/fr9Ui3yQijkXERERMlMvlTmYEAGzQsrgj4r8kfdP22h1Wb5L0b7mmAgBs\nKeuskt+U9Nn6jJIXJH0kv0gAgO1kKu6IeFrSRM5ZAAAZsHISABJDcQNAYrKOcQOaXahoZm5RS8s1\n7RkpaXpyTFPjo72OBQwcihuZzC5UdOT4SdXOrkiSKss1HTl+UpIKVd68uGAQMFSCTGbmFtdLe03t\n7Ipm5hZ7lOh8ay8uleWaQm+9uMwuVHodDegoihuZLC3X2nq+F1J4cQE6geJGJntGSm093wspvLgA\nnUBxI5PpyTGVhoc2PVcaHtL05NgWv6P7UnhxATqB4kYmU+OjOnpwn0ZHSrKk0ZGSjh7cV6gLfym8\nuACdwKwSZDY1Plqoom60lo1ZJeh3FDc2SX06XdFfXIBOoLixLpW52sCgY4wb65hOB6SB4sY6ptMB\naaC4sY7pdEAaKG6sYzodkAYuTmId0+mANFDc2ITpdEDxMVQCAImhuAEgMZmGSmx/XdJ3Ja1IOhcR\n3DgYHZP6ak2g29oZ4/7piHg5tyQYSKzWBNrHUAl6itWaQPuyFndIetT2k7YP5xkIg4XVmkD7shb3\nDRGxX9L7JX3U9o2NG9g+bHve9ny1Wu1oSPQvVmsC7ctU3BGxVP/1jKQHJB1oss2xiJiIiIlyudzZ\nlOhbrNYE2teyuG1favsda48l/aykU3kHw2BI4c46QNFkmVXyA5IesL22/d9GxCO5psJAYbUm0J6W\nxR0RL0i6pgtZAAAZMB0QABJDcQNAYihuAEgMxQ0AiaG4ASAxFDcAJIbiBoDEUNwAkBiKGwASQ3ED\nQGIobgBIDMUNAImhuAEgMRQ3ACSG4gaAxFDcAJAYihsAEkNxA0BiKG4ASAzFDQCJyXKXd0mS7SFJ\n85IqEXFrHmFmFyqamVvU0nJNe0ZKmp4c4+7fANAgc3FLukvSaUnvzCPI7EJFR46fVO3siiSpslzT\nkeMnJYnyBoANMg2V2L5C0s9J+nReQWbmFtdLe03t7Ipm5hbz+iMBIElZx7j/SNJvS/q/vIIsLdfa\neh4ABlXL4rZ9q6QzEfFki+0O2563PV+tVtsOsmek1NbzADCospxx3yDpNttfl/Q5Se+z/TeNG0XE\nsYiYiIiJcrncdpDpyTGVhoc2PVcaHtL05Fjb3wsA+lnL4o6IIxFxRUTslXRI0pcj4kOdDjI1Pqqj\nB/dpdKQkSxodKenowX1cmASABu3MKsnd1PgoRQ0ALbRV3BHxuKTHc0kCAMiElZMAkBiKGwASQ3ED\nQGIobgBIDMUNAImhuAEgMRQ3ACSG4gaAxFDcAJAYihsAEkNxA0BiKG4ASAzFDQCJobgBIDEUNwAk\nhuIGgMRQ3ACQGIobABJDcQNAYihuAEgMxQ0AiWlZ3LYvtv0V28/Yftb2J7sRDADQ3K4M27wh6X0R\n8ZrtYUn/YvvhiHgi52wAgCZaFndEhKTX6p8O1z8iz1AAgK1lGuO2PWT7aUlnJD0WESeabHPY9rzt\n+Wq12umcAIC6TMUdESsRca2kKyQdsH11k22ORcREREyUy+VO5wQA1LU1qyQiliU9LumWXNIAAFrK\nMqukbHuk/rgk6WZJz+UdDADQXJZZJZdLutf2kFaL/gsR8VC+sQAAW8kyq+Srksa7kAUAkEGWM+6B\nMrtQ0czcopaWa9ozUtL05Jimxkd7HQsA1lHcG8wuVHTk+EnVzq5IkirLNR05flKSKG8AhcF7lWww\nM7e4XtpramdXNDO32KNEAHA+inuDpeVaW88DQC9Q3BvsGSm19TwA9ALFvcH05JhKw0ObnisND2l6\ncqxHiQDgfFyc3GDtAiSzSgAUGcXdYGp8lKIGUGgMlQBAYihuAEgMxQ0AiaG4ASAxFDcAJMart5Ts\n8De1q5Je7Pg3vnC7Jb3c6xA7lPo+pJ5fSn8fUs8vpb8P2+X/oYjIdPuwXIq7aGzPR8REr3PsROr7\nkHp+Kf19SD2/lP4+dCo/QyUAkBiKGwASMyjFfazXATog9X1IPb+U/j6knl9Kfx86kn8gxrgBoJ8M\nyhk3APSNvipu27fYXrT9vO2PN/n6nbartp+uf/xyL3JuxfZnbJ+xfWqLr9v2n9T376u293c743Yy\n5H+v7Vc2/P3/brcztmL7Stv/aPu07Wdt39Vkm8Ieh4z5C30cbF9s+yu2n6nvwyebbPM225+vH4MT\ntvd2P2lzGfPvrIsioi8+JA1J+pqkd0u6SNIzkn60YZs7Jf1Zr7Nusw83Stov6dQWX/+ApIclWdJ1\nkk70OnOb+d8r6aFe52yxD5dL2l9//A5J/97k31Fhj0PG/IU+DvW/17fXHw9LOiHpuoZtfl3Sp+qP\nD0n6fK9zt5l/R13UT2fcByQ9HxEvRMSbkj4n6fYeZ2pLRPyzpO9ss8ntkv4qVj0hacT25d1J11qG\n/IUXEd+KiKfqj78r6bSkxvf5LexxyJi/0Op/r6/VPx2ufzRejLtd0r31x/dLusm2uxRxWxnz70g/\nFfeopG9u+PwlNf8H+/P1H2/vt31ld6J1TNZ9LLLr6z9CPmz7x3odZjv1H7/HtXrGtFESx2Gb/FLB\nj4PtIdtPSzoj6bGI2PIYRMQ5Sa9Iuqy7KbeWIb+0gy7qp+Ju9mrb+Cr395L2RsSPS/oHvfWKnYos\n+1hkT2l1We81kv5U0myP82zJ9tslfVHS3RHxauOXm/yWQh2HFvkLfxwiYiUirpV0haQDtq9u2KTQ\nxyBD/h11UT8V90uSNr5qXSFpaeMGEfHtiHij/ulfSPqJLmXrlJb7WGQR8eraj5AR8SVJw7Z39zjW\neWwPa7X0PhsRx5tsUujj0Cp/KsdBkiJiWdLjkm5p+NL6MbC9S9L3qoDDdFvl32kX9VNx/6uk99h+\nl+2LtHrB4sGNGzSMQ96m1fG/lDwo6Rfrsxquk/RKRHyr16Gysv2Da+OQtg9o9d/ft3ubarN6vnsk\nnY6IP9xis8Iehyz5i34cbJdtj9QflyTdLOm5hs0elPRL9cd3SPpy1K/69VqW/Dvtor6552REnLP9\nG5LmtDrD5DMR8azt35M0HxEPSvqY7dskndPqq/OdPQvchO37tHrFf7ftlyR9QqsXNhQRn5L0Ja3O\naHhe0uuSPtKbpM1lyH+HpF+zfU5STdKhovxn2+AGSR+WdLI+RilJvyPpKimJ45Alf9GPw+WS7rU9\npNUXlS9ExEMN/5fvkfTXtp/X6v/lQ72Le54s+XfURaycBIDE9NNQCQAMBIobABJDcQNAYihuAEgM\nxQ0AiaG4ASAxFDcAJIbiBoDE/D89XSUPT6XKnQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 来看看产生的x-y分布\n",
    "x, y = get_fake_data(batch_size=16)\n",
    "plt.scatter(x.squeeze().cpu().numpy(), y.squeeze().cpu().numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd8lfXd//HXFzLJIECYgRBmAjIM\nBhcVEZHhqLjq1qot991Wa9Ubxfa+7/Z3378CgnvV4mq99a5tXV3BAALiAgRRUTMIIyRhJIwkZI/z\nvf9IKMOQnJxzcq6Tc97Px4MH5HCdc3240He+fKex1iIiIl1fN6cLEBER31Cgi4gECQW6iEiQUKCL\niAQJBbqISJBQoIuIBAkFuohIkFCgi4gECQW6iEiQCPPnzRITE21KSoo/byki4jWXhQOVdZQeqcNl\nLb1jIugfH0VYN+OX+2/evPmAtbZve9f5NdBTUlLYtGmTP28pIuKxJpflzc1FPLIyl24Vddw8tj8P\nzEljRN9Yv9ZhjClw5zq/BrqISFdgrWVtXimLM3PI3X+E9OQEnr5hEpNTejtdWpsU6CIix/mquJyF\nmdl8vP0gQ/v04NkbJzFn3ACM8U/3ijcU6CIiQNHhah5ZkcfbW4rp1SOcX142lhvPGkpEWNeZO6JA\nF5GQVl7dwLNr83n5410Y4EfTRvCjaSOIjwp3urQOU6CLSEiqa2zifz4p4KnV+VTUNnBl+mDumzma\nQQnRTpfmMQW6iHRJ72wpZmlWLnvKahiUEM38WanMTU9q930ul+XvW/eyNCuHwkM1nDcqkQfnjGHs\noHg/VN25FOgi0uW8s6WYB9/aSk1DEwDFZTU8+NZWgDZDff2OgyzMzObLonLGDIznldvHM3V0u9O7\nuwwFuoh0OUuzcv8Z5kfVNDSxNCu31UDftv8Ii5fn8F5OCQN7RvHINROZm55Edz8tDPIXBbqIdDl7\nymrcer2kopbHVuXxx08LiYkI4/7Zqdw+ZRhR4d39UabfKdBFpMsZlBBNcSuhfnRAs6qukd+u28Hz\n63bQ6HJx67kp3DV9FL1jIvxdql8p0EWky5k/K/WEPnSA6PDu3HvRaF5dX8Djq7ZxoLKOS8YP5P7Z\nqQztE+Ngtf7TbqAbY14CLgVKrLXjWl5bClwG1APbgdustWWdWaiIyFFH+8mPznIZ2DOKOeMH8sza\nfHaUVjE5pRfP33IG6cm9HK7Uv4y1tu0LjJkKVAKvHBfoM4HV1tpGY8xDANbaB9q7WUZGhtXmXCLi\nS1t2H2ZRZg4bdx1ieN8YFsxO46Kx/bvEUn13GWM2W2sz2ruu3Ra6tXadMSblpNdWHPfleuDqjhYo\nIuKNgoNVLMnK5R9f7iUxNoL/P3cc104eQnj3rrNU39d80Yd+O/BHH3yOiEi7DlXV89Tqbby6voCw\nbt346YWjmDd1OLGRGhL06gkYY34BNAKvtXHNPGAeQHJysje3E5EQVtvQxMsf7eLZtflU1TVy7eQh\n/GzGaPrHRzldWsDwONCNMbfSPFh6oW2jI95auwxYBs196J7eT0RCk8tleXtLMY+syGVPeS0XpvXj\ngTlpjO4f53RpAcejQDfGzAYeAM631lb7tiQRkWYfbCtlYWYO2XsrGNIrmsSYCFbnlJCz74jbe7eE\nknZHD4wxfwA+AVKNMUXGmDuAp4E4YKUx5nNjzHOdXKeIhJDsvRXc8tJGbn5xI0dqG7j57KGUHqnj\nQFU9lmN7t7yzpdjpUgOKO7Ncrm/l5Rc7oRYRCXF7y2t4OCuPt7YUER8Vzr9fMoabzxnK9Iffp7bR\ndcK1be3dEqo0LCwijquobeC5tdt58cOdWAs/PG84P542goQezUv13d27JdQp0EXEMfWNLv53QwFP\nrs7nUFU9c08fxH0zUxnSu8cJ17W3d4s0U6CLiN9Za1n+1T6WvJvDroPVnDO8Dz+/eAzjB/ds9fpT\n7d0yf1aqv0ruEhToIuJXn+46xMLMbLbsLmN0/1he/v5kpqX2bXOp/sl7t3TkhKJQokAXEb/YXlrJ\nQ8tzWPHNfvrFRfLQVeO5+owhbh8yMTc9SQHeDgW6iHSq0iN1PPFeHn/YWEhUWDfuu2g0d5w3jB4R\nih9f0xMVkU5RXd/ICx/s5Lfvb6e20cUNZyZz94xRJMZGOl1a0FKgi4hPNbksf95UyKMr8yg5Uses\n0/pz/+w0RvSNdbq0oKdAFxGfsNayJreExctzyNtfyaTkBJ69cRIZKb2dLi1kKNBFxGtbi8pZmJnN\nJzsOktKnB7+5cRKzxw0IqkMmugIFuoh4rPBQNQ+vyOUvn++hd0wEv7psLDecNZSIsNA9ZMJJCnQR\n6bCy6nqeWZPP7z8uwBj4yQUj+JfzRxAfFe50aSFNgS4ibqtrbOKVjwt4ek0+FbUNXDVpMPfNHM3A\nnlqCHwgU6CLSLpfL8rcv97A0K5eiwzWcP7ovC+akMWZgfKvXv7OlWKs6HaBAF5E2fbz9AIsyc9ha\nXM7YgfG8escEvjMq8ZTXv7Ol+IR9V47uXQ4o1DuZAl1EWpW3/wiLl+ewOqeEQT2jePR7E5l7ehLd\n2lmqvzQr94RNtEB7l/uLAl1ETrC/opbHVubxp02FxESGsWBOGt8/N4Wo8O5uvV97lztHgS4iAFTW\nNbLs/e08/8FOGl0uvn/uMO6aPpJeMREd+hztXe4cBbpIiGtocvH6p4U8sSqPA5X1XDphIPNnpTK0\nT4xHn6e9y52jQBcJUdZaVnyzn4eW57DjQBVnpvTmhVvHcPqQBK8+V3uXO0eBLhKCPtt9mEWZ2Xy6\n6zAj+sbw/C0ZzBjTz2dL9QN573Jvp1QG8pRMBbpICNl1oIolWTlkbt1HYmwkv75iHNdmDCGse2gs\n1fd2SmWgT8lUoIuEgENV9Tz53jZeXV9AePdu3H3hKOZNHU5MZGhFgLdTKgN9SmZo/W2KhJjahiZe\n/HAnz63dTlV9I9dOTuaeGaPoFx/ldGmO8HZKZaBPyVSgiwShJpflrc+KeHRlHnvLa5kxph8L5qQx\nsl+c32sJpD5nb6dUBvqUzHY7zowxLxljSowxXx33Wm9jzEpjzLaWn3t1bpki4q7380q55MkPmP/G\nl/SLi+T1eWfzwq2THQvzB9/aSnFZDZZjfc7vbCn2ey3QPKUy+qQFUh2ZUunt+zubOyMhvwNmn/Ta\nAuA9a+0o4L2Wr0XEQV/vKefmFzdw60sbqapv5Knr03n7x1M4e3gfx2pqq8/ZCXPTk1h05XiSEqIx\nQFJCNIuuHO/2vxi8fX9na7fLxVq7zhiTctLLlwPTWn79e2At8IAP6xIRNxWX1fDIilze3lJMz+hw\n/uPSsdx0djKRYe4t1e9Mgdjn7O2UykCekulpH3p/a+1eAGvtXmNMPx/WJCJuKK9p4Ddrt/PSRzsB\nmDd1OD8+fyQ9ewTOIROB3uccbDp9UNQYMw+YB5CcnNzZtxMJevWNLl5dX8BTq7dxuLqBK9OTuHfm\naAb36uHze3k7oKltAPzL00Dfb4wZ2NI6HwiUnOpCa+0yYBlARkaG9fB+IiHPWss/tu5lybu57D5U\nzZSRfXhwzhjGJfXslPv5YhGNtgHwL08D/a/ArcDilp//4rOKRORbNu48xK8zs/misIy0AXH87rbJ\nnD+6r8+W6rfGV4toArnPOdi0G+jGmD/QPACaaIwpAn5Jc5D/yRhzB7AbuKYzixQJVfkllSxensOq\n7P30j49kydUTuGrSYLq3c8iELwTigKa0zZ1ZLtef4rcu9HEtItKi5EgtT6zaxuufFv6zz/n2KcOI\njvDfzBUNaHY9obEjj0gXUVXXyOOr8vjOQ2t4bcNumlyW2MgwkhKi/RrmEPiLaOTbtPRfJAA0Nrn4\n8+bmpfqlR+o4vkdlX0Wt1zv6eTJbRQOaXY8CXcRB1lpW55SweHkO20oqOWNoL6y1HKisP+E6b3b0\n82a2SrAMaAbSfjKdSV0uIg75sqiM65at547fb6LRZXnupkm88a/ncPCkMD/K08HIQFt+72+Btp9M\nZ1ILXcTPCg9VsyQrl799sYc+MRH81+Wncf2ZyYS3HDLh68HIUJ+tEuh7mPuSAl3ET8qq63lqdT6v\nfLKL7t0Md14wkn85fzhxUScu1ff16spQn60SSt/QFOginay2oYnff7yLZ9bkc6SukWvOGMy9F6Uy\noGfrh0z4ejAy1Jffh9I3NAW6SCdxuSx/+aKYh7PyKC6rYVpqXxbMSSNtQHy77/XlYKS/Z6sE2gBk\nKH1DU6CLdIKP8g+wMDObr/dUcNqgeJZcPYEpIxMdq8dfs1UC8RDlUJp+qUAX8aGcfRUsXp7D2txS\nkhKiefza0/nuxEF068Sl+oHUIg7UAchgmX7ZHgW6iA/sK6/l0ZW5vLG5iNjIMH5+cRq3nJNCVHjn\nru4MtBZxKA1ABiIFuogXjtQ28Nv3d/DChztocllumzKMOy8YSa+YCL/cP9BaxKE0ABmIFOgiHmho\ncvGHjbt5YtU2DlbV892Jg5g/K5UhvX1/yERbAq1FHEoDkIFIgS7SAdZasr7ex0Pv5rLzQBVnDevN\nSxePYeKQBEfqCbQWcSgNQAYiBbqImzYXHGJhZg6bCw4zsl8sL96awfS0fp16yER7ArFFHCoDkIFI\ngS7Sjp0HqnhoeQ7vfr2PvnGRLLpyPNecMZiw7s5vhaQWsRxPgS5yCgcr63jyvW28tmE3EWHduGfG\naH5w3jBiIgPrfxu1iOWowPovUyQA1NQ38dJHO/nN2u3UNDRx3eQh3D1jFP3iWl+qLxIoFOgiLZpc\nljc3F/HIylz2V9Rx0dj+PDA7jZH9Yp0uTcQtCnQJSP5c/Wit5f28UhYvzyFn3xEmDkngyevSOWt4\nn065n0hnUaBLwPHn6sevistZtDybj/IPkty7B0/fkM4l4wc6OnNFxFMKdAk4/lj9WHS4mkdW5PH2\nlmISeoTzn5eO5aazhxIR5vzMFRFPKdAl4HTm6sfymgaeXZPPyx/vAuBfzx/Bj6aNoGd0eNtvFOkC\nFOgScDpj9WNdYxP/80kBT6/Jp7ymgSvSk7hvZipJ2mNEgohXgW6MuQf4AWCBrcBt1tpaXxQmocuX\nqx9dLsvft+5laVYOhYdqOG9UIgvmpHHaoJ6+LFkkIHgc6MaYJOCnwFhrbY0x5k/AdcDvfFSbhJCj\ns1qKy2owNLcQjkqIDudX3z2tw/3n63ccZFFmNl8UlZM2II5Xbj+TqaP7elWfVmNKIPO2yyUMiDbG\nNAA9gD3elySh5uRZLfak36+qb+zQ523bf4SH3s1hVXYJA3tG8fA1E7kiPYnuHh4yEWh7jouciseB\nbq0tNsY8DOwGaoAV1toVPqtMQkZrs1qO19Bk3ZrhUlJRy2OrtvHHT3cTExHG/bNTuX3KMK8PmQi0\nPcdFTsWbLpdewOXAMKAM+LMx5iZr7asnXTcPmAeQnJzsRakSyLzpknBn9kpb11TVNbJs3Q6e/2AH\n9Y0ubjknhbumj6RPbKTb9Xtyb53CI4HGmy6XGcBOa20pgDHmLeBc4IRAt9YuA5YBZGRknPyvaQkC\n3nZJnGpWy8nXnKyxycUfNxXy2MptHKis45LxA5k/K5WUxBgP/hQdr6+z9hxXf714yptVFLuBs40x\nPUzzsroLgWzflCVdSVtdEu6YPyuV6Da6RcK7mxNmuFhrWfnNfmY9vo5fvP0VKX168NaPz+WZGyf5\nPMxPVV9n7Tl+9JtjcVkNlmPfHN/ZUuzze0nw8aYPfYMx5g3gM6AR2EJLS1xCi7ddEsfv6X3yLJde\nPcL55WXHZrh8XljGwsxsNu48xPDEGH578xnMHNu/U5fq+3PPcfXXize8muVirf0l8Esf1SJdlC+6\nJNrb07vgYBVLs3L5+5d7SYyN4L/njuO6yUMI99MhE/7ac1z99eINrRQVr3XmMWiHq+p5cvU2Xl1f\nQFi3bvx0+kjmnT+C2AA7ZKI1nvSFB9oZodK1BP7/FRLwOqNLorahiZc/2sWza/OpqmvkexlDuOei\n0fSP7xqHTHg6UByIZ4RK16FAF5/wVZeEy2V5e0sxj6zIZU95LdPT+rFgThqj+8f5oEr/8bQvXGeE\nijcU6BIwPtx2gIWZ2Xyzt4JxSfE8/L2JnDsi0emyPOJNX7jOCBVPKdDFcdl7K1i0PId1eaUkJUTz\nxHWnc9mEQXTzcKl+IFBfuDhBgS5e83QhzN7yGh5ZkcebnxURFxnGLy4ewy3nDiUyzLul+oFAfeHi\nBGOt/xZvZmRk2E2bNvntftL5Th78AwjvZoiNCqOsuqHVgK+obeC5tdt58cOdWAu3njuUn1wwkoQe\nEU78ETqNVnyKrxhjNltrM9q7Ti108Uprg38NLsvh6gbgxNkdF48fyP9uKODJ1fkcqqrn8tMH8W8z\nUxnSu4ff6/YH9YWLvynQxSvuDPLVNDTxX3/7hsdX5bHrYDXnDO/Dzy8ew/jBOmRCxJcU6OIVdzbW\nAjhUXU9iXAQvf38y01L7dupSfZFQpSPOxSvtbax1VEJ0OJk/PY8L0vopzEU6iVro4pWTF8LERYVR\nWdeI67ix9qiwbvzqu6cRdtK+Kxo0FPEttdDFa3PTk1h571TuvWg0TS1JHhPR3GpPSohm8VUTvhXU\n2iZWxPfUQhevNLksb2wu5NGVeeyvqGPm2P48MCeNEX1j23yftokV8T0FunjEWsva3FIWLc8mb38l\n6ckJPH3DJCan9Hbr/domVsT3FOjSYVuLylmYmc0nOw4ytE8Pnr1xEnPGDejQYKeWxov4ngJd3FZ4\nqJqHV+Tyl8/30Dsmgl9dNpYbzhpKRFjHh2K0NF7E9xTo0q7y6gaeWZvP7z7ahTHw42kj+NdpI4iP\nCvf4M7VNrIjvKdDllOoam3jl4wKeXpNPRW0DV00azH0zRzOwp2+6RbxZGq8pjyLfpkCXb3G5LH/7\ncg9Ls3IpOlzD1NF9WTA7jbGD4p0uDfD8NCCRYKdAlxN8sv0gi5Zn82VROWMHxvM/d4znvFF9nS7r\nBJryKNI6BboAkLf/CIuX57A6p4RBPaN49HsTmXt6UkAeMqEpjyKtU6CHuJKKWh5dmcefNhUSExnG\ngjlpfP/cFKLc2J/FKZryKNI6BXqIqqxrZNm6HTy/bgeNLhe3npvCXdNH0Tsm8A+Z0JRHkdYp0ENM\nQ5OL1z8t5IlVeRyorOeSCQO5f1YqQ/vEnPI9gTajRFMeRVrnVaAbYxKAF4BxgAVut9Z+4ovCxLes\ntaz4Zj8PvZvDjtIqzkzpzfO3pJGe3KvN9wXqjBKdBiTybd620J8A3rXWXm2MiQCC8yyxLu6z3YdZ\nlJnNp7sOM6JvDM/fksGMMe7tS64ZJSJdh8eBboyJB6YC3wew1tYD9b4pS3yh4GAVS97N5R9b95IY\nG8mvrxjHtRlDvrUveVs0o0Sk6/CmhT4cKAVeNsZMBDYDd1trq46/yBgzD5gHkJyc7MXtxF2Hqup5\n8r1tvLahgLBu3bj7wlH8cOpwYiM7/tetGSUiXYc3gR4GTALustZuMMY8ASwA/uP4i6y1y4BlABkZ\nGfZbnyI+U9vQxEsf7eQ3a7ZTVd/ItZOTuWfGKPrFR3n8ma3NKDHABWneLTYKtIFWkWDgTaAXAUXW\n2g0tX79Bc6CLnzW5LG9vKeaRFbnsLa9lxph+PDA7jVH947z+7LnpSWwqOMRr63dz9LuxBd7cXEzG\n0N4ehXCgDrSKdHUeB7q1dp8xptAYk2qtzQUuBL7xXWnijnV5pSxankP23gomDO7Jo987nXNG9Gn1\nWk9bxWtySjn5n1beDIxqoFWkc3g7y+Uu4LWWGS47gNu8L0nc8c2eChYtz+aDbQcY0juaJ69P59Lx\nA0+5VN+bVrGvB0Y10CrSObwKdGvt50CGj2oRN+wpq+GRFXm8taWI+Khw/v2SMdx8zlAiw9pequ9N\nq9jXA6MaaBXpHB0/akYcUVHbwEPv5nDBw2v525d7mHfecNbNv4AfnDe83TAH71rF82elEn3S3i7e\nLLX39eeJSDMt/Q9w9Y0uXttQwJPvbeNwdQNXpCdx38zRDO7VsTVc3rSKfb3UXkv3RTqHsdZ/Mwkz\nMjLspk2b/Ha/rsxaS+bWfSzJyqHgYDVTRvbhwTljGJfU06PPO7kPHZpbxYuuHK8gFQlwxpjN1tp2\nu7fVQg9An+46xK//kc3nhWWk9o/jd7dN5vzRfd1aqn8qahWLBD8FegDZXlrJ4uU5rPxmP/3jI1ly\n1QSuOmMw3X10yIQ2tBIJbgr0AFB6pI4n3svjDxsL/zk4ePuUYURHBO4hEyISeBToDqqub+T5dTtZ\ntm47dY0ubjwrmZ9eOIrE2EinSxORLkiB7oDGJhdvbC7i0ZV5lBypY864AcyflcrwvrFOlyYiXZgC\n3U2+2EzKWsua3BIWZeawraSSSckJ/OamSZwxtHcnVS0ioUSB7gZfbCb1ZVEZCzOzWb/jEMMSY/jN\njZOYPW6AVzNXRESOp0B3gzfL5gsPVbM0K5e/frGHPjER/Nflp3H9mcmEd+CQCRERdyjQ3eDJsvmy\n6nqeXp3PK58U0K0b3HnBSP7l/OHERYV3VpkiEuIU6G7oyLL52oYmXvlkF0+vzudIXSPXnDGYey4a\nzcCe2nhKRDqXAt0NrZ3ac/JmUi6X5a9f7GFpVi7FZTVMS+3LgjlppA2Id6JkEQlBCnQ3tLds/uP8\nAyxcns1XxRWcNiieJVdPYMrIRCdLFpEQpEB3U2vL5nP3HWHx8mzW5JaSlBDNY9dO5PKJSac8ZEJE\npDMp0D2wr7yWx1bm8efNhcREhvHgnDRuPTeFqHAt1RcR5yjQO6CyrpHfvr+d5z/YQZPLctuUYdx5\nwUh6xUQ4XZqIiALdHQ1NLl7fuJvHV23jYFU9l00cxPyZqST36dghEyIinUmB3gZrLVlf72fJuzns\nOFDFWcN689LFY5g4JMHp0kREvkWBfgqbCw6zKDObTQWHGdkvlhduyeDCMf20VF9EApYC/SQ7D1Sx\nNCuHzK376BsXycIrxvO9jMGEaam+iAQ4BXqLg5V1PLU6n1fXFxAR1o2fzRjFD88bTkykHpGIdA0h\nn1a1DU28+OFOnlu7neqGJq6dPISfzRhFv7gop0sTEekQrwPdGNMd2AQUW2sv9b4k/2hyWd76rPmQ\nib3ltcwY058Fc1IZ2S/O6dJERDziixb63UA20GU2LXk/r5RFmdnk7DvCxCEJPH7t6Zw1vI/TZYmI\neMWrQDfGDAYuAX4N3OuTijrR13vKWbw8hw+2HSC5dw+eviGdS8YP1MwVEQkK3rbQHwfuBwK6n6K4\nrIZHsnJ5+/NiekaH85+XjuXGs5OJDNNSfREJHh4HujHmUqDEWrvZGDOtjevmAfMAkpOTAd+cz+mO\n8poGnl2bz8sf7QJg3tTh/HjaSHpG65AJEQk+3rTQpwDfNcZcDEQB8caYV621Nx1/kbV2GbAMICMj\nw/rifM721De6eHV9AU+t3kZZTQNXpCdx38xUklo5kEJEJFgYa633H9LcQv+39ma5ZGRk2Mirl7R6\n+k9SQjQfLZjuVR3WWv7+5V6WZuWy+1A13xmZyII5aYxL6unV54qIOMkYs9lam9HedX6fh+7J+Zzu\n2LjzEL/OzOaLwjLSBsTx+9vPZOqoRA14ikjI8EmgW2vXAmvdubYj53O6I7+kksXLc1iVvZ8B8VEs\nvXoCV04aTHcdMiEiIcbvLXR3zud0R8mRWh5ftY0/flr4z/ffPmUY0RGauSIiocnvgd7e+Zztqapr\n5PkPdvDs2u3UN7oAiI0MIykhWmEuIiHNkb1cWjufsz2NTS7+tKmIx1blUXqkjuN7VPZV1Pp8poyI\nSFcT8HvCWmtZ9c1+Zj/xAT9/eyvJvXuQGBuB66TJOTUNTSzNynWmSBGRABDQuy1+UVjGwsxsNuw8\nxPDEGJ676Qxmndaf4Q9mtnq9tzNlRES6soAM9N0Hq1m6Ipe/fbGHPjER/Pflp3HdmcmEtxwy4euZ\nMiIiwSCgAr2sup6nVufzyie76N7NcNf0kcybOpy4qBOX6vtqpoyISDAJiECvbWji9x/v4pk1+VTW\nNXLNGUO456LRDOjZ+iET3s6UEREJRo4Gustl+csXxTyclUdxWQ0XpPZlwZwxpA5of/NGT2bKiIgE\nM8cC/aP8AyzMzObrPRWMS4pn6dUTOHdkolPliIh0eX4P9Jx9FSxensPa3FKSEqJ54rrTuWzCILpp\nqb6IiFf8GuhFh2u4+IkPiI0M4+cXp3HLOSlEhWt1p4iIL/g10Muq67lzyjDunD6ShB4R/ry1iEjQ\n82ugj+4fx79fOtaftxQRCRl+XfofERbwOw2IiHRZSlgRkSChQBcRCRIKdBGRIKFAFxEJEgp0EZEg\noUAXEQkSCnQRkSChQBcRCRIBsR+6v7yzpVh7qItI0AqZQH9nS/EJpxwVl9Xw4FtbARTqIhIUPO5y\nMcYMMcasMcZkG2O+Nsbc7cvCfG1pVu4JR9YB1DQ0sTQr16GKRER8y5sWeiNwn7X2M2NMHLDZGLPS\nWvuNj2rzqT2tHCrd1usiIl2Nxy10a+1ea+1nLb8+AmQDAdt3MSghukOvi4h0NT6Z5WKMSQHSgQ2t\n/N48Y8wmY8ym0tJSX9zOI/NnpRJ90mEa0eHdmT8r1aGKRER8y+tAN8bEAm8CP7PWVpz8+9baZdba\nDGttRt++fb29ncfmpiex6MrxJCVEY4CkhGgWXTleA6IiEjS8muVijAmnOcxfs9a+5ZuSOs/c9CQF\nuIgELW9muRjgRSDbWvuo70oSERFPeNPlMgW4GZhujPm85cfFPqpLREQ6yOMuF2vth4DxYS0iIuIF\n7eUiIhIkFOgiIkFCgS4iEiQU6CIiQUKBLiISJBToIiJBQoEuIhIkFOgiIkFCgS4iEiQU6CIiQUKB\nLiISJBToIiJBQoEuIhIkFOgiIkFCgS4iEiQU6CIiQUKBLiISJBToIiJBQoEuIhIkFOgiIkFCgS4i\nEiQU6CIiQUKBLiISJBToIiJBwqtAN8bMNsbkGmPyjTELfFWUiIh0nMeBbozpDjwDzAHGAtcbY8b6\nqjAREekYb1roZwL51tod1tp64HXgct+UJSIiHeVNoCcBhcd9XdTymoiIOCDMi/eaVl6z37rImHnA\nvJYv64wxX3lxz2CSCBxwuoiEKv1eAAADCklEQVQAoWdxjJ7FMXoWx6S6c5E3gV4EDDnu68HAnpMv\nstYuA5YBGGM2WWszvLhn0NCzOEbP4hg9i2P0LI4xxmxy5zpvulw+BUYZY4YZYyKA64C/evF5IiLi\nBY9b6NbaRmPMnUAW0B14yVr7tc8qExGRDvGmywVrbSaQ2YG3LPPmfkFGz+IYPYtj9CyO0bM4xq1n\nYaz91jimiIh0QVr6LyISJPwS6Noi4BhjzEvGmJJQn75pjBlijFljjMk2xnxtjLnb6ZqcYoyJMsZs\nNMZ80fIs/p/TNTnNGNPdGLPFGPN3p2txkjFmlzFmqzHmc3dmunR6l0vLFgF5wEU0T3X8FLjeWvtN\np944QBljpgKVwCvW2nFO1+MUY8xAYKC19jNjTBywGZgbiv9dGGMMEGOtrTTGhAMfAndba9c7XJpj\njDH3AhlAvLX2UqfrcYoxZheQYa11az6+P1ro2iLgONbadcAhp+twmrV2r7X2s5ZfHwGyCdGVxrZZ\nZcuX4S0/QnZwyxgzGLgEeMHpWroafwS6tgiQNhljUoB0YIOzlTinpYvhc6AEWGmtDdlnATwO3A+4\nnC4kAFhghTFmc8uq+zb5I9Dd2iJAQpMxJhZ4E/iZtbbC6XqcYq1tstaeTvOK6zONMSHZHWeMuRQo\nsdZudrqWADHFWjuJ5l1tf9LSZXtK/gh0t7YIkNDT0l/8JvCatfYtp+sJBNbaMmAtMNvhUpwyBfhu\nS9/x68B0Y8yrzpbkHGvtnpafS4C3ae7CPiV/BLq2CJBvaRkIfBHIttY+6nQ9TjLG9DXGJLT8OhqY\nAeQ4W5UzrLUPWmsHW2tTaM6K1dbamxwuyxHGmJiWCQMYY2KAmUCbs+M6PdCttY3A0S0CsoE/hfIW\nAcaYPwCfAKnGmCJjzB1O1+SQKcDNNLfAPm/5cbHTRTlkILDGGPMlzQ2gldbakJ6uJwD0Bz40xnwB\nbAT+Ya19t603aKWoiEiQ0EpREZEgoUAXEQkSCnQRkSChQBcRCRIKdBGRIKFAFxEJEgp0EZEgoUAX\nEQkS/wdkWEU2vxLRBQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "w:  1.9115010499954224 b:  3.044184446334839\n"
     ]
    }
   ],
   "source": [
    "# 随机初始化参数\n",
    "w = t.rand(1, 1).to(device)\n",
    "b = t.zeros(1, 1).to(device)\n",
    "\n",
    "lr =0.02 # 学习率\n",
    "\n",
    "for ii in range(500):\n",
    "    x, y = get_fake_data(batch_size=4)\n",
    "    \n",
    "    # forward：计算loss\n",
    "    y_pred = x.mm(w) + b.expand_as(y) # x@W等价于x.mm(w);for python3 only\n",
    "    loss = 0.5 * (y_pred - y) ** 2 # 均方误差\n",
    "    loss = loss.mean()\n",
    "    \n",
    "    # backward：手动计算梯度\n",
    "    dloss = 1\n",
    "    dy_pred = dloss * (y_pred - y)\n",
    "    \n",
    "    dw = x.t().mm(dy_pred)\n",
    "    db = dy_pred.sum()\n",
    "    \n",
    "    # 更新参数\n",
    "    w.sub_(lr * dw)\n",
    "    b.sub_(lr * db)\n",
    "    \n",
    "    if ii%50 ==0:\n",
    "       \n",
    "        # 画图\n",
    "        display.clear_output(wait=True)\n",
    "        x = t.arange(0, 6).view(-1, 1)\n",
    "        y = x.mm(w) + b.expand_as(x)\n",
    "        plt.plot(x.cpu().numpy(), y.cpu().numpy()) # predicted\n",
    "        \n",
    "        x2, y2 = get_fake_data(batch_size=32) \n",
    "        plt.scatter(x2.numpy(), y2.numpy()) # true data\n",
    "        \n",
    "        plt.xlim(0, 5)\n",
    "        plt.ylim(0, 13)\n",
    "        plt.show()\n",
    "        plt.pause(0.5)\n",
    "        \n",
    "print('w: ', w.item(), 'b: ', b.item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可见程序已经基本学出w=2、b=3，并且图中直线和数据已经实现较好的拟合。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "虽然上面提到了许多操作，但是只要掌握了这个例子基本上就可以了，其他的知识，读者日后遇到的时候，可以再看看这部份的内容或者查找对应文档。\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
